How to Assure the Quality of Human Computation Tasks When Majority Voting Fails?

Quality assurance remains a key topic in human computation research. Prior work indicates that independent agreement is effective for low difficulty tasks, but performs poorly for moderately difficult tasks since the majority of responses may be inaccurate. We present experimental results showing that humans are better at identifying correct answers than producing correct answers in such moderately difficult tasks. This motivates us to propose a simple quality control process based on tournament selection. We demonstrate that tournament selection repeatably finds the correct answers in as few as 3 to 5 rounds. We also compare the efficiency of the tournament selection method and elimination selection method.